System, method and computer program product for application development using a visual paradigm to combine existing data and applications

ABSTRACT

A system, method and computer program product are provided for generating an application. An interaction with a first application is recorded for capturing a functionality of the first application. An interaction with a second application is also recorded for capturing a functionality of the second application. A pattern is generated based on the recorded interactions. An application is generated based on the pattern. The interactions recorded in the pattern are repeated upon replay of the pattern for providing the functionalities of the first and second applications. A method for executing the pattern-based application is provided. Upon receiving a request for data, a pattern for retrieving the requested data is selected. The pattern is replayed. The recorded interaction is conducted as specified in the pattern for obtaining the requested data. The requested data is received.

RELATED APPLICATIONS

[0001] This application claims priority from U.S. Provisional PatentApplication serial No. 60/341,344 entitled SYSTEM, METHOD AND COMPUTERPROGRAM PRODUCT FOR APPLICATION DEVELOPMENT USING A VISUAL PARADIGM TOCOMBINE EXISTING DATA AND APPLICATIONS, filed Dec. 14, 2001; U.S. PatentApplication entitled SYSTEM, METHOD AND COMPUTER PROGRAM PRODUCT FOR THERECORDING AND PLAYBACK OF TRANSACTION MACROS, filed non-provisionallyAug. 28, 2001 under Ser. No. 09/942,047; U.S. Patent Applicationentitled SYSTEM, METHOD AND COMPUTER PROGRAM PRODUCT FOR PATTERN REPLAYUSING STATE RECOGNITION, filed non-provisionally Aug. 28, 2001 underSer. No. 09/942,080; and U.S. Patent Application entitled INTERFACE FORMOBILIZING CONTENT AND TRANSACTIONS ON MULTIPLE CLASSES OF DEVICES,filed non-provisionally Sep. 14, 2001 under Ser. No. 09/953,372; each ofwhich is assigned to common assignee Clickmarks, Inc., and hereinincorporated by reference.

FIELD OF THE INVENTION

[0002] The present invention relates to computer programming, and moreparticularly to creating new applications using functionality fromexisting applications.

BACKGROUND OF THE INVENTION

[0003] The term application is a shorter form of application program. Anapplication program is a program designed to perform a specific functiondirectly for the user or, in some cases, for another applicationprogram. Examples of applications include word processors, databaseprograms, Web browsers, development tools, drawing, paint, image editingprograms, and communication programs. Applications use the services ofthe computer's operating system and other supporting applications. Theformal requests and means of communicating with other programs that anapplication program uses is called the application program interface(API).

[0004] In computing, a program is a specific set of ordered operationsfor a computer to perform. In the modem computer that John von Neumannoutlined in 1945, the program contains a one-at-a-time sequence ofinstructions that the computer follows. Typically, the program is putinto a storage area accessible to the computer. The computer gets oneinstruction and performs it and then gets the next instruction. Thestorage area or memory can also contain the data that the instructionoperates on. (Note that a program is also a special kind of “data” thattells how to operate on “application or user data.”)

[0005] Programs can be characterized as interactive or batch in terms ofwhat drives them and how continuously they run. An interactive programreceives data from an interactive user (or possibly from anotherprogram). A batch program runs and does its work, and then stops. Batchprograms can be started by interactive users who request theirinteractive program to run the batch program. A command interpreter or aWeb browser is an example of an interactive program. A program thatcomputes and prints out a company payroll is an example of a batchprogram. Print jobs are also batch programs.

[0006] When a program is created, it is written using some kind ofcomputer language. The language statements are the source program. Thedeveloper then “compiles” the source program (with a special programcalled a language compiler) and the result is called an object program(not to be confused with object-oriented programming). There are severalsynonyms for object program, including object module and compiledprogram. The object program contains the string of 0s and 1s calledmachine language that the logic processor works with. The machinelanguage of the computer is constructed by the language compiler with anunderstanding of the computer's logic architecture, including the set ofpossible computer instructions and the length (number of bits) in aninstruction.

[0007] From the above, it should be apparent that it takes weeks tocreate an application, even for a skilled programmer. Consider thefollowing example. Company A want to develop an enterprise travelapplication. The application is to be constantly running, monitoring ahuman resources database for details on when executives are traveling,then going into a travel site (such as Travelocity.com, etc.) andchecking flight times, checking confirmations, etc. The application isalso to access a rental car site and make reservations. Upon completingthe aforementioned tasks, the application is to allow each employee tocheck the status of his or her itinerary. The application is also tosend alerts to the manager and/or administrative assistant of theemployee regarding the travel arrangements.

[0008] To create such an application, a developer has to program theapplication to access several types of databases and gather data,process the data, and send the processed data out to potentially severaltypes of devices (PC, phone, PDA, Blackberry, etc.). One skilled in theart will appreciate the time that must be spent when writing the codefor such an application, ensuring that all API's are properly coded to,etc.

[0009] What is needed is a method of creating an application quickly andefficiently without requiring the typical weeks and months of coding anddebugging.

[0010] Another problem inherent in the prior art is that mostapplications are created for the desktop. In an enterprise situation, aclient-server implementation is most typical. However, only one clientcan interact with a particular session of the software. As modernbusinesses expand, they will grow to rely on enterprise-wideapplications. Thus, pervasive computing will become the rule.

[0011] What is needed is a way to provide pervasive (vs. desktop)applications that are accessible throughout an enterprise on varioustypes of devices and via multiple modes of connection.

[0012] Web services (sometimes called application services) are services(usually including some combination of programming and data, butpossibly including human resources as well) that are made available froma business's Web server for Web users or other Web-connected programs.Providers of Web services are generally known as application serviceproviders. Web services range from such major services as storagemanagement and customer relationship management (CRM) down to much morelimited services such as the furnishing of a stock quote and thechecking of bids for an auction item. The accelerating creation andavailability of these services is a major Web trend.

[0013] Users can access some Web services through a peer-to-peerarrangement rather than by going to a central server. Some services cancommunicate with other services and this exchange of procedures and datais generally enabled by a class of software known as middleware.Services previously possible only with the older standardized serviceknown as Electronic Data Interchange (EDI) increasingly are likely tobecome Web services. Besides the standardization and wide availabilityto users and businesses of the Internet itself, Web services are alsoincreasingly enabled by the use of the Extensible Markup Language (XML)as a means of standardizing data formats and exchanging data. XML is thefoundation for the Web Services Description Language (WSDL).

[0014] Several companies provide the building blocks for creating Webservices. However, one problem inherent in creating Web services is thata developer is limited by the number of building blocks he or she hasaccess to. In other words, the developer has a “fixed menu” of buildingblocks, and must write code for any additional functionality desired.

[0015] What is needed is a methodology for reusing exiting functionalityrather than having to create custom applications for each desiredfunction. Such a methodology would allow companies to leverage existinginfrastructure, thereby saving money.

SUMMARY OF THE INVENTION

[0016] A developer is able to stitch together components of existingapplications, including web applications, web services, legacyapplications, enterprise applications, mainframe applications, WINDOWS®applications, and a JAVA® applications and is also able to add newapplication functionality using a variety of programming languagesincluding Java, JavaScript and SQL, all using a visual paradigm thatallows the application to be built very quickly. Using an intuitive userinterface, the developer can publish the new application as a webservice. The newly developed application can be made accessible onmultiple devices via any type of network, wireless or Internetconnection.

[0017] This process allows the application to be created quickly withoutrequiring the developer to spend days and weeks writing code for a newapplication. Rather, using the processes set forth herein, selectedportions of applications in an existing infrastructure can be reused toprovide new functionality and applications. The developer simplyinteracts with the existing applications and those interactions arerecorded to capture the resulting functionality.

[0018] Accordingly, a system, method and computer program product areprovided for generating an application. An interaction with a firstapplication, preferably at the user-interface level, is recorded forcapturing a functionality of the first application. An interaction witha second application is also recorded for capturing a functionality ofthe second application. Note that the interactions may be userinteractions, interactions by other applications, etc. A pattern isgenerated based on the recorded interactions. The pattern can be savedin memory such as in a file on a hard disk. An application is generatedbased on the pattern. The interactions recorded in the pattern arerepeated upon replay of the pattern for providing the functionalities ofthe first and second applications.

[0019] This allows the application to be created quickly withoutrequiring the developer to spend weeks and months writing code for a newapplication. Rather, using the process set forth above and below,selected portions of an existing infrastructure can be reused to providenew functionality and applications. The developer simply interacts withthe existing applications and those interactions are recorded.

[0020] According to one embodiment, additional functionality can beadded to the pattern. The pattern can be displayed on a graphical userinterface, where the pattern includes a graphical representation of eachinteraction. According to another embodiment, when recording theinteraction with one or more of the applications, an interface can bedisplayed. The user is allowed to specify interactions with theapplication(s) via the interface. The user interactions are recorded andbecome part of the pattern.

[0021] The application can be published as a network service accessiblevia the Internet, an intranet, or other network. Further, a rendering ofthe output from the application can be specified for transcoding andoutput on multiple types of devices the content can be rendered foroutput on multiple types of devices such as a desktop computer, awireless telephone, a PDA, and/or any other type of device.

[0022] A method for creating an application utilizing a graphical userinterface is also provided. First, an interface is displayed. A user isallowed to specify interactions with multiple applications via theinterface. These interactions are recorded. A pattern is generated basedon the recorded interactions. The pattern is displayed on a graphicaluser interface, and the pattern includes a graphical representation ofeach interaction. Such graphical representation can be an icon, etc. Theuser is allowed to add additional functionality to the pattern. Agraphical representation of the additional functionality is alsodisplayed on the graphical user interface.

[0023] Preferably, the user is allowed to connect representations of theinteractions via the graphical user interface for specifying an order ofexecution of the interactions in the pattern. As an option, the patterncan include prerecorded interactions. Thus, previously recordedinteractions can be used as “building blocks” for the new application,thereby avoiding the need to re-record the interaction. They are merelyretrieved from storage and added to the pattern to providefunctionality.

[0024] According to an embodiment, if a portion of the pattern isrecognized as a prefix of an already existing pattern, a branching ofthe pattern is created from the already existing pattern. Also, thepattern can be coupled to another pattern to provide additionalfunctionality. The other pattern can be a previously created patternthat has been stored. Preferably, output relating to the pattern can bespecified for multiple client devices.

[0025] Once the application has been created, the pattern-basedapplication can be executed. Upon receiving a request for data from auser or another application, a pattern for retrieving the requested datais selected based on stored information associating the pattern with thedesired functionality required to obtain the requested data. The patternhas stored therein a recorded interaction with at least one externalapplication. The interaction accesses the functionality of the externalapplication, thereby permitting data to be obtained from the externalapplication. The pattern is replayed to execute the recordedinteraction. The recorded interaction is conducted as specified in thepattern for obtaining the requested data. The requested data is receivedand can be further processed and/or output.

[0026] The received data can then be processed for output on aparticular type of device as specified in the pattern, and/or can beprocessed for output on multiple types of devices via transcoding.

BRIEF DESCRIPTION OF THE DRAWINGS

[0027]FIG. 1 illustrates a system including a user and a system, inaccordance with one embodiment;

[0028]FIG. 2 illustrates a method for carrying out acomputer-implemented transaction;

[0029]FIG. 2A illustrates an exemplary flow diagram showing informationthat may be stored with the transaction pattern;

[0030]FIG. 2B illustrates an exemplary flow diagram showing the mannerin which a transaction pattern may be executed;

[0031]FIG. 3 shows a representative hardware environment on which themethod of FIG. 2 may be implemented;

[0032]FIG. 4 illustrates an exemplary flow diagram illustrating thevarious components of a first exemplary transaction pattern;

[0033]FIG. 5 illustrates an exemplary flow diagram illustrating thevarious components of a second exemplary transaction pattern;

[0034]FIG. 6 illustrates an exemplary flow diagram illustrating thevarious components of a third exemplary transaction pattern;

[0035] FIGS. 7-9 illustrate an example of dynamic content in the contextof an embodiment;

[0036]FIG. 10 is a flow chart that illustrates a method for recording apattern using the User Agent according to one embodiment;

[0037]FIG. 11 illustrates a recording interface according to oneembodiment;

[0038]FIG. 12 illustrates a mode selection screen of the interface ofFIG. 11;

[0039]FIG. 13 shows a field selection screen of the interface of FIG.11;

[0040]FIG. 14 depicts a filter screen of the interface of FIG. 11;

[0041]FIG. 15 illustrates a results screen of the interface of FIG. 11;

[0042]FIG. 16 illustrates an assign variable screen of the interface ofFIG. 11;

[0043]FIG. 17 shows the grid screen of the interface of FIG. 11;

[0044]FIG. 18 shows the grid screen of FIG. 17 with additional objectsadded;

[0045]FIG. 19 illustrates a display specification screen of theinterface of FIG. 11 from which the user can specify the manner in whichvariables are displayed on particular devices;

[0046]FIG. 20 illustrates an exemplary system for replaying a patternusing a network, in accordance with one embodiment;

[0047]FIG. 21 is a drawing showing a general overview of how a patternreplay engine interfaces with other major components of the systemaccording to one embodiment;

[0048]FIG. 22 is a flowchart of a transcoding process for renderingarbitrary content for display on a particular viewing device accordingto a preferred embodiment;

[0049]FIG. 23 is a flowchart depicting generation of new applicationsaccording to one embodiment;

[0050]FIG. 24 is a flow diagram of a process for generating anapplication according to one embodiment;

[0051]FIG. 25 is a flow diagram depicting a process for creating anapplication utilizing a graphical user interface in accordance with anembodiment;

[0052]FIG. 26 provides a flow of execution of the pattern-basedapplication; and

[0053]FIG. 27 is a system architecture diagram according to oneembodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0054]FIG. 1 illustrates a system 100 including a user 102 and a system104, in accordance with one embodiment of the present invention. Duringuse, the user 102 and the system 104 interact during transactions. Inthe present invention, a transaction refers to communicating (i)information and/or actions required to conduct the transaction, and/or(ii) information and/or actions sent back or desired by the user,respectively.

[0055] For example, a transaction, in one embodiment, may refer to:information submitted by the user, actions taken by the user, actionstaken by a system enabling the access of the user to the data, actionstaken by the data to retrieve/modify content, results sent back to theuser, and/or any combination or portion of the foregoing entities.

[0056]FIG. 2 illustrates a method 200 for carrying out acomputer-implemented transaction. Initially, in operation 202 atransaction pattern reflecting a transaction is stored in memory. Thismay be carried out while the transaction is happening.

[0057] In one embodiment of the present invention, the transactionpattern may include a record of: information submitted by a user,actions taken by the user, actions taken by a system to generateresults, and results sent to the user. As an option, the transactionpattern may include a record of actions taken by the system which enableaccess of the user to data, and actions enabled by the data to retrievecontent. Storage in accordance with operation 202 may take any form withsufficient identification and description of any required step in thetransaction process.

[0058]FIG. 2A illustrates an exemplary flow diagram 220 illustrating thevarious information which may be stored during operation 202 of FIG. 2.For example, such information may involve (i) the web or non-webinterfaces presented to the user, (ii) the actions performed by the useror the system in the submission of the information, (iii) the parametersrequired to complete the transaction and submitted by the user, (iv) theautomatic or manual navigation of the user within the transactionprocess, and/or (v) the content or results returned by the system orselected by the user.

[0059] During use of the present invention, such transaction pattern maybe retrieved from memory when desired in operation 204. Thereafter, thetransaction pattern is executed to carry out the transaction in anautomated manner. Note operation 206. An execution, or playback, inaccordance with operation 206 may include the invocation of a storedtransaction pattern.

[0060]FIG. 2B illustrates an exemplary flow diagram 230 illustrating thevarious operations that may occur during execution of a transactionpattern in accordance with operation 206 of FIG. 2. Such may involve:(i) a direct or indirect retrieval of the transaction pattern by anautomated and/or programmable agent, i.e. a computer server/client, anactive or dormant program, a human being, etc. (ii) an automatic or userprompted submission of the required parameters in every step of thetransaction process, (iii) the automatic navigation within thetransaction process, (iv) a retrieval of the content returned by thesystem or specified by the user, and/or (v) the relaying of the contentback to the user.

[0061] A transaction pattern for any transaction need only be recordedonce by anyone with sufficient system access privilege. Once recorded,any user may conduct that transaction, any number of times, by theinvocation of the recorded transaction pattern, or “transaction macro.”

[0062] Since a transaction macro can be stored in any fashion andinvoked, directly or indirectly, by any agent (human or automated), itenables the completion of the transaction it describes in any fashionfrom any device.

System Architecture

[0063]FIG. 3 shows a representative hardware environment on which themethod 200 of FIG. 2 may be implemented. Such figure illustrates atypical hardware configuration of a workstation in accordance with apreferred embodiment having a central processing unit 310, such as amicroprocessor, and a number of other units interconnected via a systembus 312.

[0064] The workstation shown in FIG. 3 includes a Random Access Memory(RAM) 314, Read Only Memory (ROM) 316, an I/O adapter 318 for connectingperipheral devices such as disk storage units 320 to the bus 312, a userinterface adapter 322 for connecting a keyboard 324, a mouse 326, aspeaker 328, a microphone 332, and/or other user interface devices suchas a touch screen (not shown) to the bus 312, communication adapter 334for connecting the workstation to a communication network 335 (e.g., adata processing network) and a display adapter 336 for connecting thebus 312 to a display device 338.

[0065] The workstation typically has resident thereon an operatingsystem such as the Microsoft Windows NT or Windows 2000 Operating System(OS), the IBM OS/2 operating system, the MAC OS, or UNIX operatingsystem. Those skilled in the art may appreciate that the presentinvention may also be implemented on platforms and operating systemsother than those mentioned.

Applications

[0066] Following is a plurality of exemplary applications of the presentinvention. It should be noted that such examples are not exhaustive, andshould not be construed as limiting.

Scenario I

[0067]FIG. 4 illustrates an exemplary flow diagram 400 illustrating thevarious components of a first exemplary transaction pattern. Duringoperation, a user wishes to retrieve a record from a database. Theretrieval of the record involves submitting the required parameters intoan interface presented to the user for interacting with the database.Once the parameters are submitted, the results are sent back to the userin some predetermined manner.

[0068] In this case, the transaction pattern includes: (i) requiredparameters submitted by the user (ii) the entire internal process ofsubmission of these parameters to the database by the interface and(iii) the results returned by the database and presented to the user.Note FIG. 4.

Scenario II

[0069]FIG. 5 illustrates an exemplary flow diagram 500 illustrating thevarious components of a second exemplary transaction pattern. Duringoperation, a user wishes to purchase an item from an e-commerce website. The user interacts with the e-commerce interface presented on theweb site and submits the required information to purchase the item. Theuser potentially fills out multiple forms (in multiple steps) thatrequest the user's name, credit card information, shipping address, etc.Once the information has been entered, it is processed and the purchaseis approved or rejected.

[0070] In this case, the transaction pattern consists of (i) thecreation and actions associated with the forms presented in theweb-interface with which the user submits information (ii) theinformation submitted by the user, in every form in every step of thee-commerce flow (iii) the internal process whereby the submittedinformation is sent to the servers and databases of the e-commerce site(iv) the navigation (automatic or user determined) of the user withinthe e-commerce process and (v) the results returned by the e-commercesite once the submitted information has been processed. Note FIG. 5.

Scenario III

[0071]FIG. 6 illustrates an exemplary flow diagram 600 illustrating thevarious components of a third exemplary transaction pattern. Duringoperation, a user wishes to log into his stock portfolio account andview his portfolio. He supplies his username, password and potentiallyother information to gain secure access to this personal information.Once he is logged in, he wishes to view a table summarizing theperformance of his stocks. To get to this table, he has to click on aseries of hyperlinks, one after the other.

[0072] In this case, the transaction pattern consists of (i) thecreation and the action associated with the forms presented in theweb-interface with which the user submits information (ii) theinformation submitted by the user, in every form in every step of thelogin and account access process (iii) the internal process whereby thesubmitted information is sent to the servers and databases of the user'sportfolio account site (iv) the navigation (automatic or userdetermined) of the user within the portfolio account and (v) all thecontent that the user desires in every step of his navigation within theportfolio account. Note FIG. 6.

[0073] In all of these scenarios, the user submits and retrievesdifferent pieces of information in multiple steps. While it is possibleto submit information in this fashion from a desktop computer, itbecomes increasingly inconvenient once a user tries to do so throughdevices other than PC's where the entry of information is not asconvenient. Therefore, a process whereby a query or a transaction can becompleted and the resulting content sent to the user with a minimumamount of information input and navigation, is highly desirable.

[0074] The present invention thus allows any transaction to be automatedthrough the concept of the recording and the playback of a “transactionmacro.”

Dynamic Content

[0075] A description of dynamic content in the context of the presentinvention will now be set forth along with a comparison with staticcontent. A computer network system has one or more host network serversconnected to serve data to one or more client computers over a network.FIG. 7 shows a simple computer network system 700 with a single hostnetwork server 702 connected to serve data to a client 704 via a network706. The client 704 sends a request for data and/or services to theserver 702 over the network 706. The server 702 processes the requestand returns a response over the network 706. If the request is for data,the server 702 accesses a database 708 to retrieve the requested data710 and returns the data 710 as part of the response.

[0076] The client-server system 700 is representative of many differentenvironments. One particular environment of interest is the Internet.The server 702 runs a Web server software program that accepts requestsfrom client-based programs (e.g., browsers) and returns data 710 in theform of Web pages or documents to the client 704. The Web pages arecommonly written in HTML (hypertext markup language) and XML (extensiblemarkup language). Web pages are transmitted using conventional networkprotocols, such as TCP/IP (Transmission Control Protocol/InternetProtocol), HTTP (Hypertext Transfer Protocol) and DCOM (DistributedComponent Object Model). The client 704 executes a browser or otherapplication to render the Web page into human-perceptible forms. A Webdocument might include text, images, sound, video, active code, and soforth.

[0077] Documents served from a server to client are typically generatedusing either or both of two methods: a static content method and adynamic content method. In a static content method, the document iscreated in advance and stored statically on a server database. When aclient requests the document, the server retrieves the document andtransmits it over the network to the client. FIG. 7 is an example inwhich the server retrieves the static data 710 from database 708 andserves the data to the client 704. It is further noted that conventionalservers, and particularly Web servers, may be configured to push thecontent to the client without receiving a request. The static contentmethod has an advantage of minimizing the user's perceived responsetime, meaning the time between requesting the document and seeing itrendered on a computer screen. It has a disadvantage that all users whorequest the document receive exactly the same content. With staticcontent, the server cannot respond to specific user requests orpersonalize a document for individual users.

[0078] In a dynamic content method, the document is generateddynamically by the server. When a client requests a document, the serverinvokes one or more agents, feeding the agents relevant parameters fromthe user's request (such as the user's name). The agent(s) generate thedocument that satisfies the user's request and the server returns thedocument over the network to the client. The dynamic content method hasthe advantage of responding to specific user requests or personalizingcontent for individual users. It has the disadvantage that the user'sperceived response time will generally be longer than with staticdocument requests. This is because the document generation processinvolves additional time to invoke the appropriate agent(s) and generatethe document.

[0079] The server generates dynamic content documents by invoking anagent in one of two ways: an “out-of-process” method and an “in-process”method. In an “out-of-process” method, the agent runs in its own processand address space, separate from the server's process and address space.Typically, the out-of-process method uses the industry-standard commongateway interface (CGI) as the communication mechanism between theserver and agent. In an “in-process” method, the agent runs within theWeb server's process and address space. The in-process method typicallyuses a vendor-specific application programming interface, like theInternet Server Application Programming Interface (ISAPI) implemented byInternet Information Server (IIS), which is available from MicrosoftCorporation.

[0080] To illustrate the two dynamic content methods and how they can beused in conjunction with the static content method, consider a scenarioin which the server 702 runs a Web server for an online retail company.When the client 704 first accesses the Web site, the server 702 mightretrieve a pre-existing home page for the company from the database 708and serve that page to the client 704. This initial step is an exampleof a static content method. From the home page, the client might requestto view an online catalog of products offered by the company. Inresponse, the Web server might invoke a catalog agent to guide the userthrough various product offerings. When the user decides to purchase aproduct, the client submits an order request. In response, the Webserver might invoke an order agent to assist the user in ordering theproduct. The steps involved with actively serving a catalog or taking anorder are examples of dynamic content methods. They both involve dynamicgeneration of documents in response to input received from the client.

[0081]FIG. 8 shows an “out-of-process” method under this scenario. Theserver 702 runs a Web server 800 as process 1. The Web server 800handles the incoming requests from the client. When the client firsthits the Web site, the Web server 800 retrieves the company's home page802 from the database 708 and transmits the home page 802 to the client.When the client sends an order request, the Web server 800 initiates anorder manager 804 to assist the user with ordering the desired productor service. The order manager 804 is initiated using the CGI technologyas a second process 2, which uses a separate process and address spacethan process 1, as represented by the dashed lines. When the userselects an item, the order manager 804 dynamically generates an orderdocument 806 that contains the user's name, a description of theselected item, the cost of the item, and payment terms. The ordermanager 804 returns the order document 806 to the Web server 800, whichthen serves the document 806 to the client. Afterwards, the ordermanager 804 is terminated and the second process 2 is halted.

[0082] The out-of-process method shown in FIG. 8 has an advantage incrash prevention and recovery. If the out-of-process order manager 804is unreliable and ultimately crashes, it will not cause the Web server800 to crash. However, the out-of-process method has a disadvantage inthat a particular agent must be loaded into memory each time a requestarrives for it. Using CGI technology, the agent must also be unloadedfrom memory once it finishes the request. This loading and unloadingconsumes resources, resulting in a relatively slow response time.Another problem compounding the slowness is that the out-of-processmethod involves cross-process communication between processes 1 and 2,including such activities as marshalling, messaging, and the like.

[0083] A variation of CGI, known as FastCGI, allows the server to keepthe agent loaded, rather than terminating the agent each time itresponds to a particular request. FastCGI is an improvement over CGI inthat it saves the per-request invocation overhead, thereby improving theresponse time. However, the FastCGI is still run in a separate process,and hence the drawbacks associated with cross-process communicationremain.

[0084]FIG. 9 shows an “in-process” method under the same scenario of theonline retail company. In this case, when the client sends an orderrequest, the Web server 800 initiates an order manager 902 to assist theuser with ordering the desired product or service. The order manager 902is initiated using the ISAPI technology to run within the same process 1as the Web server 800. That is, the Web server 800 and the order manager902 use the same process and address space designated by the server.When the user selects an item from the online catalog, the order manager902 dynamically generates an order document 904 that contains the user'sname, a description of the selected item, the cost of the item, andpayment terms. The order manager 902 returns the order document 904 tothe Web server 800, which then serves the document 806 to the client.The order manager 902 can remain loaded for other requests, orterminated when desired.

[0085] The in-process method is advantageous in that a particular agentneed only be loaded into memory once, when the first request arrives forit. The agent can then remain in memory within the server's process andaddress space, and can handle additional requests that arrive for it.The in-process method has a further advantage in that there is no needfor cross-process communication. As a result, the in-process method iscomparatively faster than the out-of-process method.

Pattern Recording

[0086] A preferred embodiment of the present invention gives its usersthe ability to create (record) functional elements called patterns anduse those elements (replay) in their custom applications. A patternconsists of a collection of states. A state refers to the state of anapplication in its interaction by the user (the designer). States can beof different types. For example, an action that requires accessing theWorld Wide Web can be represented by an XML (or other type) state and anaction that would require sending e-mail can be a part of an SMTP state.Each type of state has a dedicated connector, which is used to createthe state definitions and operate on those states. Thus, an XMLConnector Module (XCM) is the connector that aids in recording andreplaying of XML states. A database connector is used to store andretrieve data from a database, but may or may not point to a next state.An HTML Connector points to a next table. For example, the HTMLConnector in state A would instruct a User Agent to execute action X toaccess state C. The User Agent is used to fetch the output from a remoteapplication and execute any user actions on that output.

[0087]FIG. 10 is a flow chart 1000 that illustrates a method forrecording a pattern using the User Agent. In particular, the userperforms the desired navigation and associated input, which arerecorded. In operation 1002, the user is asked for a URL. Upon receivingthe URL from the user, the URL is sent to the User Agent in operation1004. The User Agent parses the request in operation 1006 to determine ameaning of the request. In operation 1008, the User Agent retrieves theweb page associated with the URL and sends it back to the user. This(and other) web pages are represented in the pattern as States. Inoperation 1010, actions are received from the user. Such actions caninclude selection of links, input of data in fields, etc. The User Agentreceives the user input in operation 1012, and in operation 1014,executes the actions on the remote website. The actions are recorded inoperation 1016. The resulting web page is shown the to the user inoperation 1018. The user then identifies the parts of the page he or sheis interested in, which is received in operation 1020. The identifiedparts are recorded in operation 1022. Note that the inputs entered bythe user during recording can be generalized by associating a variablewith those inputs, which can be changed during replay.

[0088] The system now knows how to retrieve desired information usingthe pattern. Accordingly, all users using the system can obtain thatcontent. The next (or same) user that wants to perform a similar actionenters the URL of the pattern by selecting it. The pattern is replayedwith the User Agent executing the actions received of operation 1014.The parts of the web page that were identified by the user and stored inoperation 1022 are returned to the user. The user can also define adefault area to access in the event that a state is not recognized. Seealso the description below regarding state identification based oncontent and probability.

[0089] The information stored in a pattern may involve (i) the web ornon-web interfaces presented to the user, (ii) the actions performed bythe user or the system in the submission of the information, (iii) theparameters required to complete the transaction and submitted by theuser, (iv) the automatic or manual navigation of the user within thetransaction process, and/or (v) the content or results returned by thesystem or selected by the user.

[0090] A transaction pattern for any transaction need only be recordedonce by anyone with sufficient system access privilege. Once recorded,any user may conduct that transaction, any number of times, by theinvocation of the recorded transaction pattern, or “transaction macro.”

[0091] Since a transaction pattern can be stored in any fashion andinvoked, directly or indirectly, by any agent (human or automated), itenables the completion of the transaction it describes in any fashionfrom any device.

Interactive Design Tool (IDT)

[0092] The Interactive Design Tool according to one embodiment allows auser to mobilize content and transactions on to multiple classes ofdevices. Some specific examples of the use of the IDT include:

[0093] Mobilize an existing web application (manipulate the data foroutput on a mobile device).

[0094] Create a new mobile application based partially on existingcontent from the web, a database etc. (For example, giving a salespersonthe ability to mark an order as closed from a mobile device resulting ina legacy database system being updated)

[0095] Create a Web Integration application based on the interaction ofmultiple existing web applications.

[0096] Configure the basic view presented by the mobilization program(i.e. editing the application layout schema). Note that this lastfeature is conceptually different from the first three as it operates ata “higher” level; it defines the containers within which the variousapplications defined in the first three cases above may appear.

Application Views

[0097] The IDT graphical user interface provides several views accordingto one embodiment.

[0098] Workplaces

[0099] The first view is the Workplace view. Workplace views are theviews within which the designer specifies the content and/or transactionby example. For example, while mobilizing web content, the Web Workplacepresents a browser view to the designer where the designer can indicatespecific content by clicking on the areas of interest. The workplacerecords the usage pattern in the background. If the designer ismobilizing content stored in a database, the Database Workplace displaysthe appropriate database connection dialogs. Thus, the workplace is atall times, aware of the class of content/transactions the designer ismobilizing and provides the appropriate view. The designer accessescontent with multiple types of connectors. Connectors are functionalblocks that allow access via various methods such as Web, databaseconnectivity, FTP, etc.

[0100] Pattern View

[0101] The Pattern View is a view where the designer is able tomanipulate the properties of the pattern. The Pattern View can alsoallow the designer to create a pattern manually and/or add patternelements directly. Several functions enabled by the Pattern View follow.

[0102] The Pattern View allows formal pattern specification. Especiallyfor web patterns, the pattern recorded in the Workplace represents onlyone possible interaction path. For example, the pattern may onlyrepresent a successful login into an online account. However, in actualusage, a user may also encounter an unsuccessful login. The Pattern Viewis used to visually specify such alternate paths in a flowchart format.Note that formal specification may be needed not just for the webconnector. For example, some connectors may require error handlingalternate interaction paths. See the section on Recording AlternatePaths, below, for further details.

[0103] The Pattern View also enables cross-connector flow, includingdesign of complex interactions that span multiple types of connectors.For example, consider an interaction that obtains content like stockquotes from the web and inserts them into a database. The two pieces ofthis interaction (web and database) are individually recorded within theWorkplace. However, the Pattern View is where the information flowacross these pieces is coupled, or “glued,” together.

[0104] A pattern may be divided into multiple segments. For example, apattern which aims to check on the price using a price comparison engineand then purchase the item, might be first divided up into two segments(one for the comparison and one for the purchase). The designer caneventually link the individual segments together while applyingappropriate selection and transformation using scripts.

[0105] Device Editor View(s)

[0106] The Device Editor View(s) is a view where the designer is able tospecify the view and interaction of the Client Devices as it relates tothe pattern under design. For example, consider a pattern for retrievingbank account information. Within the Workplace and Pattern View, thedesigner is able to specify how to fetch the account information whilenavigating through multiple web screens. In the Device Editor, thedesigner may specify, for example, two screens: one in which the bankaccount number is entered and a second where the account balance isdisplayed. The Device Editor View is able to accommodate design formultiple classes of devices (e.g. phones, PDAs, etc.). Further, theDevice Editor View(s) can function as the screen editors for specificclients, and can depict an accurate rendering of what the client devicedisplays.

[0107] Script Editor View

[0108] Within the Script View, the designer is able to edit Javascriptto manipulate the data flow within the pattern and also serve as gluinglogic.

[0109] Utility Views

[0110] These are views used to enhance the user experience and provideuseful information at various points. One Utility View is a ProjectExplorer. The Project Explorer displays all elements defined in theproject thus far in a tree format. The main branches of the tree caninclude: Pattern (which contains all the states defined thus far in thePattern View grouped into segments), Scripts (all scripts defined),and/or Session Data.

[0111] Another Utility View is a Data Explorer. Using the Data Explorer,the designer is able to create and inspect the data variables. Thesevariables can be created for a give pattern and/or can be predefined andprovided in a database of the mobilization program. The variables can beof basic data types but also can belong to User Defined data types thatcan also be created via this View. These variables are available to beused in different views.

[0112] A further Utility View is a Pattern Gallery, which displays alist of available patterns that can be re-used. Various properties ofthe patterns (inputs, outputs, description, last modification date,etc.) can be shown.

[0113] Replay Mode (Testing and Debugging)

[0114] According to one embodiment, once a pattern has been recorded,the IDT can be used to replay the pattern. This is critical for testingand debugging the pattern. The designer will have the ability to setbreakpoints, step through the pattern and set up variable watch windowsto trace the execution of the pattern. Since the pattern is interpreted,it may be possible to stop at a breakpoint, change the pattern that issupposed to follow, and re-start execution of the pattern.

[0115] IDT Output

[0116] According to an embodiment, the IDT can generate XML descriptionsof the pattern called Pattern XML (or other suitable name), the mobileviews generated and the data transformations, etc. designed within thedata view. In addition, the scripts written by the designer are alsorecorded. The IDT also stores the current project in a binary document(uses MFC object serialization mechanisms), which has the information ofthe Pattern defined so far. The project file can be called “CompanyProcess Document” (*.cpd files) or other suitable name.

[0117] More information about the IDT is provided in copending U.S.Patent Application entitled INTERFACE FOR MOBILIZING CONTENT ANDTRANSACTIONS ON MULTIPLE CLASSES OF DEVICES, from which priority isclaimed.

[0118]FIG. 11 illustrates a recording interface 1100. As shown, a gridscreen 1102 is provided. It is here that the graphical representation ofan application for retrieving data from a data site is displayed. AnObject Bar 1104 allows insertion of connectors and shapes into the gridscreen. A Catalog Bar 1106 displays several types of specializedconnectors and components. One specialized connector is the databaseconnector 1108 representing a connection to a database. Selection of thetabs on the Catalog Bar displays different connectors and components.The items in these bars can be dragged and dropped into the grid screenand connected with connectors from the Object Bar. The Project Bar 1110shows the names and types of variables being manipulated. Variables canbe added by using a wizard, which starts upon selection of the wizardbutton 1112.

[0119] The following example illustrates creation of an application forretrieving a customer identifier (ID) from a database and creating aninterface for displaying the customer ID on a particular type and/ormodel of device. As will be discussed in the example, the recordinginterface allows creation of different interfaces for different devicesbased on the capabilities of each.

[0120] Referring again to FIG. 11, a start screen 1114 is displayed uponselection of the wizard button. The database server is selected from thedrop down menu 1116 and the Next button is selected. A databaseselection screen (not shown) is presented, which allows the user toselect the desired database from which the desired data is to beretrieved.

[0121]FIG. 12 illustrates a mode selection screen 1200. Here, the useris allowed to select step by step mode or advanced mode. The desiredmode is chosen and the Next button is selected. In this example, thestep by step mode is shown.

[0122]FIG. 13 shows a field selection screen 1300. The tables of theselected database are presented in the drop down menu 1302. The user isallowed to select which table to view. The fields of the selected tableare presented in the fields display portion 1304. The user is allowed toselect one or more fields from the field display portion. Note, multiplefields can be chosen using CTRL+R-click. The Next button is selected.

[0123]FIG. 14 depicts a filter screen 1400, which allows entry offiltering elements used to restrict the variables retrieved from thedatabase. As shown, the field chosen in the field selection screen isdisplayed in the Field column 1402. Operands can be entered into theOperator column 1404 (or chosen from a drop down menu). Illustrativeoperands include >, <, ≧, ≦, and =. The Value column 1406 receivescomparison values, to which the variables associated with the field arecompared using the operand. The Next button is selected.

[0124]FIG. 15 illustrates a results screen 1500 displaying the resultsof the query in a results portion 1502. The query is also displayed in aquery portion 1504. The query may be edited in the query portion tochange the results returned. The Execute button 1506 is selected tore-execute the query and retrieve results of the edited query. The Nextbutton is selected.

[0125]FIG. 16 illustrates an assign variable screen 1600. Here, theresults are assigned to a variable and saved for later use. The name ofthe variable is entered in the Variable Name field 1602. In thisexample, the variable is named “customer.” The Next button is selected.

[0126]FIG. 17 shows the grid screen 1100. The new variable “customer”now appears in the Project Bar 1110. A Database Connector 1702 isdragged from the Catalog Bar, as is a Select Query object 1704. Aconnector 1706 is dragged and dropped from the Object Bar to the gridscreen and used to connect the Database Connector and the Select Queryobject.

[0127]FIG. 18 shows the grid screen with additional objects added. Here,a Start Process object 1802, an Output object 1804, an End Processobject 1806, and connectors have been added.

[0128]FIG. 19 illustrates a display specification screen 1900 from whichthe user can specify the manner in which variables are displayed onparticular devices. As shown, various devices are shown in a treestructure. A type of device is selected, here the Palm V version of thePDA. A representation 1902 of the PDA is displayed. The identifier“customer” 1904 is added to the simulated display 1906 of the PDA. Thevariable field 1908 is also added to the display. During runtime, theactual result of the query is displayed in place of the identifier andvariable field. Text may also be added for output in this screen. Here,the words “Customer ID:” 1910 have been added.

[0129] For example, a database program may have a table having an outputwith five columns. On a PDA, perhaps only four columns can be displayedon the display. The output can be set to show only four variables.Similarly, if only two columns of the table can be displayed on a phone,the output can be set to show only two variables.

Alternate Paths

[0130] The ability to specify alternate paths is especially useful forthe XML Connector. The returned page may or may not be the ‘expected’page, i.e. the page that the designer saw during recording, or it may besome different page. For example, during recording, the designer goes toa URL which is a login page, enters a Login/Password and can see thecontents. During pattern replay, it is possible that the returned pageafter this interaction is a ‘Server too busy’ page. In such a situation,the designer may now want to retry for the password entry page.

[0131] To handle such a situation, the designer records the normal pathusing XML Connector Workplace and switches to Pattern View. From the‘Palette View’, the designer drags and drops the XML Connector icon onthe Pattern View. The designer links the login page node with this newconnector node. The designer right-clicks the new connector node andselects ‘Identify Page’ from a pop-up menu. This invokes the XMLConnector Workplace. The designer now browses to the alternate page andclicks ‘identify’. Details for this identify operation are provided inU.S. Patent Application entitled SYSTEM, METHOD AND COMPUTER PROGRAMPRODUCT FOR RECOGNIZING A STATE OF A REMOTE APPLICATION, filed Aug. 28,2001 under Ser. No. 09/942,263.

[0132] The workplace recording allows the designer to follow a givenpath in a linear fashion. Consider the scenario, of the XML Connector,where in a certain state there are choices for interactions (i.e. somelinks to choose from) that results in different outcomes. The designerdesires to cover the alternate paths too. Even if the workplace onlyallowed recording in a linear fashion of a single scenario, the IDTwould recognize a sub-pattern that is the “prefix” of an alreadyrecorded pattern, and if the designer revisits the workplace and startsrecording again to reach the same state that matches a previouslyrecorded pattern and then branches off to an alternate path, theworkplace would still show a linear recording but the IDT wouldrecognize the prefix and eventually insert the branching off from theprefix in the Pattern View.

Illustrative System Architecture for Pattern Replay

[0133]FIG. 20 illustrates an exemplary system 2000 for replaying apattern using a network 2002, in accordance with one embodiment of thepresent invention.

[0134] A Request Handler (RH) 2004 communicates with a user device 2005.The RH manages requests from the user device, routing them to theappropriate system component. When a user requests a transaction, therequest is sent to a Pattern Replay Engine (PRE) 2006, which replays apattern for conducting one or more transactions with other applications2020, 2022 to provide functionality. More information about the PRE isset forth below.

[0135] The State Recognition Module (SRM) 2008 determines which state awebsite is in based on its current output, such as a structure of thecurrent output. The SRM may communicate with a Content RecognitionModule 2010, which recognizes individual documents that can comprise astate.

[0136] A Connector 2012 is in communication with the SRM. The Connectorexecutes a state in the pattern.

[0137] The User Agent 2014 is used by other components of the system toprovide the actual interaction with a remote application. For example,when replaying a pattern, the SRM communicates with the User Agent viathe Connector to provide instructions to the User Agent. The othersystem components have intelligence built into them that instructs themhow to utilize the User Agent. For example, when a user clicks on abutton on a page, other components instruct the User Agent to navigateto the desired application and perform some action, such as filling in aform. The User Agent retrieves the resulting output from the applicationand returns it to the other components.

[0138] By default, the User Agent is not running. A listener (not shown)listens for requests. When the listener receives a request, it creates anew User Agent process on the server and returns an identifier thatidentifies the User Agent process. Subsequently, client processes usethe identifier, go to the specific User Agent and instruct it to performsome action. The User Agent performs the action according to theinstructions and returns the results of the action.

[0139] Content is retrieved from a remote application and is sent to aspecific device in a format suitable for that device. The formatting isdone by another module, namely, the Transcoding Page Rendering Engine(TRE). A Content Retrieval Module (CRM) (not shown) retrieves thecontent as an XML stream for use by the other system components.Components that directly use the CRM are the State Recognition Module(SRM) and the Interactive Development Tool (IDT). More information aboutoperation and functionality of the Content Retrieval Module is found incopending U.S. Patent Application entitled SYSTEM, METHOD AND COMPUTERPROGRAM PRODUCT FOR RETRIEVING ARBITRARY CONTENT FROM A REMOTE DATA SITEFOR OUTPUT TO A DEVICE IN A FORMAT SUITABLE FOR THAT DEVICE, filed Aug.28, 2001 under Ser. No. 09/942,262, which is herein incorporated byreference.

[0140] The Transcoding Page Rending Engine (TRE) 2016 renders contentfor display on the user device. Preferably, the TRE is able to rendercontent on any display environment.

Pattern Replay Engine (PRE) 2006

[0141] The Pattern Replay Engine (PRE) is a component that isresponsible for replaying a pattern from one particular state to anotherduring runtime.

[0142]FIG. 21 gives a general overview of how the PRE 2006 interfaceswith other major components including the Input Handler 2104, sessionmanager 2106, connector 2108, and TRE 2110.

[0143] The PRE is a request-driven component. The PRE takes the requestto make a transition from a particular State. Transitions from one Stateto another are made by executing Actions. The PRE communicates with theappropriate Connector (of the current State) to execute the currentAction in the request and get the new resulting State. Next, all Scriptassociated with the State is evaluated.

[0144] The PRE then calls the Transcoding Page Rendering Engine (TRE) to“flush” its output to the device. This means that any Content, which hasbeen queued for display in the TRE (while processing the request), issent to the device. Every state can potentially generate output to bedisplayed on the device, including internal states. This is a desirablefeature as it means the designer can send a status message to thedevice, e.g. “Please wait while processing..” etc. Finally, a script iscalled, which can initiate further State transitions from this scriptmethod if necessary.

[0145] More information about operation and functionality of the PRE isfound in U.S. Patent Application entitled SYSTEM, METHOD AND COMPUTERPROGRAM PRODUCT FOR PATTERN REPLAY USING STATE RECOGNITION, from whichpriority is claimed.

Transcoding Page Rendering Engine (TRE) 2016

[0146] The Transcoding Page Rendering Engine (TRE) is a tool used torender content on any display environment.

[0147]FIG. 22 is a flowchart of the TRE process 2200 for renderingarbitrary content for display on a particular viewing device accordingto a preferred embodiment of the present invention. In operation 2202,content is received. In operation 2204, the content is built into a DOMtree in an extended version of XHTML, referred to herein as CXHTML. Notethat CXHTML is used herein by way of example only, and any suitableformat can be used. For example, the format can be any XML form. ThisDOM tree is then processed in operation 2206 by several modules whichtranslate the CXHTML into an appropriate markup language for the viewingenvironment and format this markup language to display best on theviewing device in operation 2208, which may include splitting the markuplanguage into multiple pages. Note operation 2210. The markup languagerepresentation of the content can also be translated into the user'snative language and/or character set. In operation 2212, the content isoutput to the viewing device.

[0148] Note that the TRE can process other types of output, such asaudio, etc.

[0149] More information about operation and functionality of the TRE isfound in U.S. Patent Application entitled SYSTEM, METHOD AND COMPUTERPROGRAM PRODUCT FOR PAGE RENDERING UTILIZING TRANSCODING, filednon-provisionally Aug. 28, 2001 under Ser. No. 09/942,051 and assignedto common assignee Clickmarks, hic., and which is herein incorporated byreference.

Pervasive Computing Through Interaction Level Programming

[0150] Interaction Level Programming (ILP)—Interaction Level Programmingis the act of creating a software application, using the user-interface(UI) of another program as the Application Program Interface (API).Because many applications that do not expose an API, do expose a UI, ILPallows many more applications to be re-factored into e.g. mobileapplications, web applications, and web services. ILP effectively turnsthe application UI into an API.

[0151] Visual Interaction Level Programming (VILP)—Visual ILP is the actof visually creating a program using the UI of another application. Theprogrammer interacts with the application via the application's userinterface as they normally would. In addition, the programmer visuallyinstructs the ILP system what actions to carry out, and what UI elementsto extract from the application. The user's interactions are recorded bya tool such as the IDT described above. The recorded interactions maythen be replayed.

[0152] When replayed by the ILP system, the resulting program carriesout the specified actions against the application's user interface, andexposes the specified UI elements to the program for use in its userinterface.

[0153] For clarity, the present discussion will refer primarily to ILP,it being understood that the discussion applies equally to VILP.

[0154] The new application created via ILP can expose differentinterfaces, including user interfaces, and/or programming interfaces(API), allowing the new application to be used in completely newenvironments and computing systems. For example, ILP could be applied toan IBM 3270 mainframe application, to create a new application thatallows mobile users to enter information into the 3270 application fromtheir web-enabled cell phones. All this is done without changing theoriginal 3270 application in any way whatsoever, thus providing completeflexibility in integrating different applications together withouthaving to modify said applications.

Finite State Machine

[0155] According to a preferred embodiment, ILP works by modeling theapplication as a finite state machine (FSM), where each state is the UIpresented by the application at a given time. Transitions between statesare made by performing a sequence of actions on the UI, known as anevent stream. The exact definition of a state in the FSM is determinedby the ILP programmer.

[0156] A state in the FSM is defined by the UI elements present in theUI of the application at a given point in time. The presence of UIelements can be given different weights, thus allowing slightlydifferent UI screens to be treated as the same state by the ILP system.This flexibility allows the user to simplify the FSM, as it does notneed to contain as many states as there are different UI screens. Forexample, two UI screens that differ only in background color can berepresented as the same state in the FSM.

State Recognition

[0157] The ILP system recognizes at runtime what state in the FSM theapplication is actually in. The state is recognized using a heuristicsearch that computes the recognition score of each possible state basedon the weighted presence or absence of UI elements in the application.For example, suppose that after a given transition, the FSM could be inpotentially one of 2 states. In one state, a certain UI element, say abutton, must be present for the FSM to be in that state. In the otherstate, that same button must not be present. Thus the system evaluatesthe recognition score for each state, and picks the best one, in thiscase, depending on whether or not the button is present.

[0158] State recognition is important, because it allows for tremendousflexibility in constructing the FSM. For example, suppose that in acertain state, the user is expected to enter a text value into an entryfield, and then press a submit button. 99% of the time, the applicationends up in state A upon pressing the button. But if the user enters acertain value in the entry field, the application ends up in state Binstead. These states may be very different from each other. Forexample, consider a simple login form in a typical web application. Ifthe user enters the correct user name and password she is taken to apage displaying some personal information. If not, the user is promptedto try again for a few times, and after that is told that they mustcontact technical support. In all cases the user carries out the sameactions (enter username and password, press submit) but ends up in 3potentially different states. Because the logic of which state the userwill end up is embedded in the application, there is no easy way toexpress this fact when constructing an external model of theapplication. Instead, the model simply indicates that pressing thesubmit button in the login state can lead to 3 possible states. Which ofthe states was actually reached is determined at runtime by the ILPsystem.

Performing Actions on a State

[0159] Once the system determines what FSM state it is in, it carriesout the actions defined for that state, and if the actions result in astate transition, it again recognizes the FSM state, and so on. Theactions that can be performed on a given state are:

[0160] Send UI events to the application (e.g. push button 1, enter avalue into field 2, etc.). Extract UI elements from the application(e.g. extract an HTML table from a website). Create UI widgets tied backto the application (e.g. create a form widget that corresponds to a formin the application).

[0161] Together, these actions allow the user of the ILP system tocreate a new application or “program” that derives part or all of itsfunctionality from the original application.

[0162]FIG. 23 is a flowchart depicting generation of new applications2302 according to one embodiment. Using the IDT described above and thevarious system components set forth in FIG. 20 and the relateddiscussion, a developer is able to stitch together components 2304 ofexisting applications 2306, including web applications, web services,legacy applications, enterprise applications, mainframe applications,WINDOWS® applications, and a JAVA® applications and is also able to addnew application functionality using a variety of programming languagesincluding Java, JavaScript and SQL, all using a visual paradigm thatallows the application to be built very quickly. Using an intuitive userinterface, the developer can publish the new application as a webservice, accessible via HTTP, Simple Object Access Protocol(SOAP—described below), etc. through a centralized server. The newlydeveloped application can be made accessible on multiple devices 2308via any type of network, wireless or Internet connection.

[0163] This process allows the application to be created quickly withoutrequiring the developer to spend days and weeks writing code for a newapplication. Rather, using the processes set forth herein, selectedportions of applications in an existing infrastructure can be reused toprovide new functionality and applications. The developer simplyinteracts with the existing applications and those interactions arerecorded to capture the resulting functionality. The developer no longerneeds to code to API's. Rather, the methodology presented herein goesabove API's, to the interaction level, thus making it easier to call thefunctionality of the existing infrastructure.

[0164]FIG. 24 is a flow diagram of a process 2400 for generating anapplication via ILP according to one embodiment. An interaction with afirst application, preferably at the user-interface level, is recordedin operation 2402 for capturing a functionality of the firstapplication. The applications can be any type of application inexistence. In operation 2404, an interaction with a second applicationis also recorded for capturing a functionality of the secondapplication. Note that the interactions may be user interactions,interactions by other applications, etc. A pattern is generated inoperation 2406 based on the recorded interactions. The pattern can besaved in memory such as in a file on a hard disk.

[0165] When recording the interaction with one or more of theapplications, an interface can be displayed to allow a user to specifyinteractions with the application(s) via the interface. The userinteractions are recorded and become part of the pattern as the userinteracts.

[0166] In operation 2408, an application is generated based on thepattern. The interactions recorded in the pattern are repeated uponreplay of the pattern for providing the functionalities of the first andsecond applications. In optional operation 2410, the application ispublished as a network service accessible via the Internet, an intranet,or other network.

[0167]FIG. 25 is a flow diagram depicting a process 2500 for creating anapplication via VILP utilizing a graphical user interface. In operation2502, an interface is displayed. A user is allowed to specifyinteractions with multiple applications via the interface in operation2504. These interactions are recorded in operation 2506. In operation2508, a pattern is generated based on the recorded interactions. Thepattern is displayed on a graphical user interface in operation 2510.The pattern includes a graphical representation of each interaction.Such graphical representation can be an icon, etc. See FIGS. 11-19 andrelated discussion for more details regarding graphical representations.In operation 2512, the user is allowed to connect representations of theinteractions via the graphical user interface for specifying an order ofexecution of the interactions in the pattern.

[0168] With continued reference to FIG. 25, the user is allowed to addadditional functionality to the pattern in operation 2514. In operation2516, a graphical representation of the additional functionality isdisplayed on the graphical user interface. Thus, by creating a patternwith recorded interactions and additional functionality, a developer hascreated an application that can be used and reused.

[0169] As an option, the pattern can include prerecorded interactions.Thus, previously recorded interactions can be used as “building blocks”for the new application, thereby avoiding the need to re-record theinteraction. They are merely retrieved from storage and added to thepattern to provide functionality.

[0170] As another option, the pattern can be coupled to another patternto provide additional functionality. The other pattern can be apreviously created pattern that has been stored or another patterncreated jointly.

[0171] Preferably, output relating to the pattern can be specified formultiple client devices. This can include specifying which output isactually sent to a particular type of device, as well as the format ofthe output for particular types of devices.

[0172] Once the application has been created, the pattern-basedapplication can be executed. FIG. 26 provides a flow of execution 2600of the pattern-based application. Upon receiving a request for data froma user or another application in operation 2602, a pattern forretrieving the requested data is selected in operation 2604 based onstored information associating the pattern with the desiredfunctionality required to obtain the requested data. The pattern hasstored therein a recorded interaction with at least one externalapplication. The interaction accesses the functionality of the externalapplication, thereby permitting data to be obtained from the externalapplication. In operation 2606, the pattern is replayed to execute therecorded interaction. In operation 2608, the recorded interaction isconducted as specified in the pattern for obtaining the requested data.The requested data is received in operation 2610. The received data canbe further processed and/or output. Note operations 2612 and 2614. Forexample, the received data can then be processed for output on aparticular type of device as specified in the pattern, and/or can beprocessed for output on multiple types of devices such as a desktopcomputer, a wireless telephone, a PDA, and/or any other type of device.This can be done by transcoding the output as described above.

EXAMPLE

[0173] Consider the following example of a logistics application. Acompany is currently using the following system. A truck driver deliversgoods and notes the delivery on a sheet, returns to headquarters, andgives sheet to data entry. If there were problems or he was unable todeliver the goods, he writes the details on the sheet and data entrysends an email to a manager.

[0174] The company wants to automate this process. The desiredapplication is to have the following parameters. The driver is to begiven a PDA or voice phone (for communicating via a voice portal, asdescribed in copending U.S. Patent Application entitled VOICEAPPLICATIONS AND VOICE-BASED INTERFACE, filed Sep. 14, 2001 under Ser.No. 09/952,909, and which is herein incorporated by reference). Thedriver is to enter a status or problem via the PDA or voice phone. Theinformation is either entered in database, or sent to report generation,which creates a report and sends it to a manager. The manager is to beable to check status/reports on a desktop.

[0175] To perform all of these functions, the application must interfacewith a backend resource management system (how company resources arebeing used), and go into a customer management system (customerdatabase) to see if the customer is not available at a particulardelivery time or has billing issues. The application must also be ableto communicate with the driver's device, whether connected ordisconnected. Also, the application must be capable of generating analert to the manager.

[0176] To create such a pervasive application in the prior art method,the developer must learn about and write code for the PDA (Palm Pilot,etc). Code must also be written to handle the transactions that occurwhen the driver interacts and his data comes to the application on theserver. A connection to the databases must be opened and the driver'scommunications written in. Further, the developer must write code thatchecks for exceptions written by someone (for example, if deliverydoesn't happen by this time, send alert) via an administrative interfacecreated by someone. Then if someone (manager) wants to log on through adesktop, the developer must create an interface for that too. Thus, thedeveloper has to write a lot of software.

[0177] According to the methodology presented herein, a pervasiveapplication connecting all of these applications and devices can bequickly built and implemented. Using the IDT, the developer merely needsto interact with the various devices and programs. The IDT captures theinteractions, which are then used to create a pattern or patterns.

[0178] A form presented on the PDA can be generated using the publisherfor mobile devices described in U.S. Patent Application entitled SYSTEM,METHOD AND COMPUTER PROGRAM PRODUCT FOR A CONTENT PUBLISHER FOR WIRELESSDEVICES filed Jul. 10, 2001 under Ser. No. 09/902,929, and which isherein incorporated by reference. The driver can enter information onthe form, which is transmitted to the new application via an existingcommunications program. The developer interacts with the existingcommunications program in such a way that the desired data is obtained.The developer's interaction is recorded and placed in a pattern, asdescribed above. Upon replay of the pattern, the desired data is againretrieved. See FIGS. 11-19 for a description of capturing interactionswith databases. To create the alerts, the developer interacts with anemail program, entering delivery and content information. Again, thisinteraction is recorded and placed in the pattern. The developer is thenable to set criteria for which alert is sent upon receipt of informationfrom the driver.

Transaction Object Model (TOM)

[0179] One embodiment allows developers to create transaction objects,which they can use as building blocks to create applications. In sharpcontrast to the prior art, the methodology set forth herein overcome the“fixed menu” drawback of prior art systems and allows a developer to useany existing functionality by capturing it in a pattern.

[0180] Any software application can be seen as a set of functionalflows. Each transaction can be seen as a transaction object. Thetransaction objects are captured and converted into building blocks thata developer can use and reuse. The developer no longer needs to selectfrom a predefined set of building blocks, but can capture the blocksfrom any application. Now, the entire universe of applications providesbuilding blocks. Any portion of any application by anyone can become abuilding block.

[0181] The developer selects the desired blocks and writes some codearound the block (if necessary) to create a new application. Further,the system is flexible, allowing the developer to add and remove blocksat will.

[0182] Simple Object Access Protocol (SOAP) is a way for a programrunning in one kind of operating system (such as Windows 2000) tocommunicate with a program in the same or another kind of an operatingsystem (such as Linux) by using the World Wide Web's Hypertext TransferProtocol (HTTP) and its Extensible Markup Language (XML) as themechanisms for information exchange. Since Web protocols are installedand available for use by all major operating system platforms, HTTP andXML provide an already at-hand solution to the problem of how programsrunning under different operating systems in a network can communicatewith each other. SOAP specifies exactly how to encode an HTTP header andan XML file so that a program in one computer can call a program inanother computer and pass it information. It also specifies how thecalled program can return a response.

[0183] SOAP was developed by Microsoft, DevelopMentor, and UserlandSoftware and has been proposed as a standard interface to the InternetEngineering Task Force (IETF). It is somewhat similar to the InternetInter-ORB Protocol (IIOP), a protocol that is part of the Common ObjectRequest Broker Architecture (CORBA). Sun Microsystems' Remote MethodInvocation (RMI) is a similar client/server interprogram protocolbetween programs written in Java.

[0184] An advantage of SOAP is that program calls are much more likelyto get through firewall servers that screen out requests other thanthose for known applications (through the designated port mechanism).Since HTTP requests are usually allowed through firewalls, programsusing SOAP to communicate can be sure that they can communicate withprograms anywhere.

System Architecture

[0185]FIG. 27 is a diagram of a system architecture 2700 according to apreferred embodiment. The system according to this embodiment includesthe following components, each of which is discussed in greater detailbelow. One component is a pool of Application Instances 2702. Theserepresent the application being programmed. Another component is anInstance Manager 2704, which manages a potentially re-usable pool ofapplication instances. A Ul Converter 2706 translates the current stateof the application's UI into an internal representation for processing.An Application Server 2708 receives instructions to record at designtime, and replays these instructions at runtime. A Visual Tool 2710renders the application UI so that the designer can visually program theapplication.

[0186] Application Instances 2702

[0187] Each application instance represents an interactive session,generally corresponding to a single user of the ILP application. Notethat in client-server applications, only multiple client instances areneeded. Each application instance can be a pattern, as described above.

[0188] Instance Manager 2704

[0189] The Instance Manager (IM) allows the system to scale to a largenumber of simultaneous users. For each user interaction, the systemrequests an application instance from the IM. Because applicationinstances may be expensive (generally an entire operating system processis required for each instance) the IM may implement instance pooling,where application instances are re-used across users. The IM preferablyseparates instances so that instances do not pollute each other.

[0190] UI Converter 2706

[0191] The UI Converter (UIC) takes a “snapshot,” of the application'sUI state, and converts it to a representation that is used for staterecognition. The “snapshot” can be an identification of selected itemsand/or features of the UI (UI elements), a graphical image of thescreen, the table structure of the UI state, etc.

[0192] Application Server 2708

[0193] The Application Server carries out the ILP application's logic,including state recognition, and performing the specified actions ateach state. The application server also serves as a platform foraccessing the ILP application via e.g. HTTP, SOAP, etc. Thus, theapplication server can include a state recognition module (describedabove) and a pattern replay engine (described above).

[0194] Visual Tool 2710

[0195] The Visual Tool allows the use of VILP, where the programming isdone visually by interacting with the application. The user can visuallydesignate states and actions, and record events as they are performedagainst the application to be replayed at runtime. In a preferredembodiment, the visual tool is an IDT, mentioned above.

Scalability

[0196] Because ILP accesses the application through the user interface,a large amount of system resources are potentially used up by each ILPuser session. Note however, that the current model for distributedcomputing is that each user has a fairly powerful desktop machine thatruns the client, while a high-powered server processes client requests.Thus a worst-case scenario might require that for each ILP user sessiona single desktop workstation is required to run the application. In factthe ILP system allows multiple user sessions to be created on a singleserver machine. So, in fact, ILP is more scalable than currentdistributed computing practices. Compared to the cost of eventhin-client terminals such as PDAs, the cost of using ILP to enablethese new client terminals to access existing applications is small. Ifa $1000 server can support 10 simultaneous users, that is a cost of $100per simultaneous user. The actual cost per user is even lower unless allusers are always simultaneously accessing the system. Compared to alow-end PDA, which would cost $100, this is minimal. And compared to thedevelopment costs of replacing the application, or integrating it withother applications at the source code level, ILP is a very favorablechoice.

[0197] While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. Thus, the breadth and scope of a preferred embodiment shouldnot be limited by any of the above-described exemplary embodiments, butshould be defined only in accordance with the following claims and theirequivalents.

What is claimed is:
 1. A method for generating an application,comprising: recording an interaction with a first application forcapturing a functionality of the first application; recording aninteraction with a second application for capturing a functionality ofthe second application; generating a pattern based on the recordedinteractions; and generating a new application based on the pattern,wherein the interactions recorded in the pattern are repeated uponreplay of the pattern for providing the functionalities of the first andsecond applications.
 2. The method as recited in claim 1, wherein theinteraction with the first application is at a user-interface level ofthe first application.
 3. The method as recited in claim 1, wherein thefirst and second applications are selected from a group consisting of: aweb application, a web service, a database application, a legacyapplication, a mainframe application, a WINDOWS® application, and aJAVA® application.
 4. The method as recited in claim 1, whereingenerating the new application includes adding additional functionalityto the pattern.
 5. The method as recited in claim 1, further comprisingdisplaying the pattern on a graphical user interface, wherein thepattern includes a graphical representation of each interaction.
 6. Themethod as recited in claim 1, wherein recording the interaction with thefirst application includes: displaying an interface; allowing a user tospecify interactions with the first application via the interface; andrecording the user interactions.
 7. The method as recited in claim 1,further comprising publishing the new application as a network service.8. The method as recited in claim 1, further comprising specifying arendering of output from the new application for output on multipletypes of devices.
 9. A computer program product for generating anapplication, comprising: computer code for recording an interaction witha first application for capturing a functionality of the firstapplication; computer code for recording an interaction with a secondapplication for capturing a functionality of the second application;computer code for generating a pattern based on the recordedinteractions; and computer code for generating an application based onthe pattern, wherein the interactions recorded in the pattern arerepeated upon replay of the pattern for providing the functionalities ofthe first and second applications.
 10. A system for generating anapplication, comprising: logic for recording an interaction with a firstapplication for capturing a functionality of the first application;logic for recording an interaction with a second application forcapturing a functionality of the second application; logic forgenerating a pattern based on the recorded interactions; and logic forgenerating an application based on the pattern, wherein the interactionsrecorded in the pattern are repeated upon replay of the pattern forproviding the functionalities of the first and second applications. 11.A method for creating an application utilizing a graphical userinterface, comprising: displaying an interface; allowing a user tospecify interactions with multiple applications via the interface;recording the interactions; generating a pattern based on the recordedinteractions; displaying the pattern on a graphical user interface,wherein the pattern includes a graphical representation of eachinteraction; allowing the user to add additional functionality to thepattern, wherein a graphical representation of the additionalfunctionality is displayed on the graphical user interface.
 12. Themethod as recited in claim 11, further comprising allowing the user toconnect representations of the interactions via the graphical userinterface for specifying an order of execution of the interactions inthe pattern.
 13. The method as recited in claim 11, wherein the patternfurther includes prerecorded interactions.
 14. The method as recited inclaim 11, further comprising recognizing a portion of the pattern as aprefix of an already existing pattern, and creating a branching of thepattern from the already existing pattern.
 15. The method as recited inclaim 11, further comprising coupling the pattern to another pattern.16. The method as recited in claim 11, further comprising allowingspecification of output relating to the pattern for multiple clientdevices.
 17. A method for executing a pattern-based application,comprising: receiving a request for data; selecting a pattern forretrieving the requested data, the pattern having stored therein arecorded interaction with at least one external application, theinteraction being for obtaining data; replaying the pattern; conductingthe recorded interaction as specified in the pattern for obtaining therequested data; and receiving the requested data.
 18. The method asrecited in claim 17, further comprising processing the received data foroutput on a particular type of device as specified in the pattern. 19.The method as recited in claim 17, further comprising processing thereceived data for output on multiple types of devices.
 20. A method forexecuting a pervasive application generated from other applications,comprising: determining a current state of an existing application basedon user interface output of the application; performing an actionpre-defined for the current state of the existing application; andretrieving data from the existing application, wherein the data isgenerated in response to the performance of the action.
 21. The methodas recited in claim 20, wherein the state is determined based onelements of the user interface output of the existing application. 22.The method as recited in claim 21, wherein some elements of the userinterface output have a different weighting than other elements forallowing other user interface output data to be treated as the samestate.
 23. The method as recited in claim 21, wherein the state isdetermined based on a computation of a recognition score of eachpossible state of the existing application based on the weightedpresence of user interface elements in the current user interfaceoutput.
 24. The method as recited in claim 20, further comprisingrepeating the method if performing the action results in a statetransition.
 25. The method as recited in claim 20, wherein theapplication is selected from a group consisting of: a web application, aweb service, a database application, a legacy application, a mainframeapplication, a WINDOWS® application, and a JAVA® application.
 26. Asystem for creation and playback of an application, comprising aplurality of application instances, each application instancerepresenting an application; an instance manager for managing theapplication instances; a state recognizer for recognizing a state of anexisting application based on user interface output of the existingapplication; an application server for performing interactions with theexisting application at a particular state; and a visual tool forrecording interaction with the existing application for later replay bythe application server.
 27. The system as recited in claim 26, furthercomprising a converter for converting user interface output of theexisting application to another format for processing.
 28. A method forcreating and executing an application utilizing a graphical userinterface, comprising: displaying an interface; allowing a user tospecify interactions with multiple applications via the interface;wherein the interaction with a first application is at a user-interfacelevel of the first application; recording the interactions; generating apattern based on the recorded interactions; displaying the pattern on agraphical user interface, wherein the pattern includes a graphicalrepresentation of each interaction; allowing the user to add additionalfunctionality to the pattern, wherein a graphical representation of theadditional functionality is displayed on the graphical user interface;allowing the user to connect representations of the interactions via thegraphical user interface for specifying an order of execution of theinteractions in the pattern; recognizing a portion of the pattern as aprefix of an already existing pattern, and creating a branching of thepattern from the already existing pattern; coupling the pattern toanother pattern; allowing specification of output relating to thepattern for multiple client devices; saving the pattern in memory;wherein the pattern further includes prerecorded interactions;generating an application based on the pattern, wherein the interactionsrecorded in the pattern are repeated upon replay of the pattern forproviding the functionalities of the applications; publishing theapplication as a network service; receiving a request for data;selecting a pattern for retrieving the requested data, the patternhaving stored therein a recorded interaction with at least one externalapplication, the interaction being for obtaining data; replaying thepattern; conducting the recorded interaction as specified in the patternfor obtaining the requested data; receiving the requested data;processing the received data for output on multiple types of devices;and outputting the data.